book

Spark: The Definitive Guide

by Bill Chambers, Matei Zaharia

February 2018

Intermediate to advanced

606 pages

14h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Has Sandbox

About the AuthorsWho This Book Is ForConventions Used in This BookUsing Code ExamplesO’Reilly SafariHow to Contact UsAcknowledgments
Apache Spark’s PhilosophyContext: The Big Data ProblemHistory of SparkThe Present and Future of SparkRunning SparkDownloading Spark LocallyLaunching Spark’s Interactive ConsolesRunning Spark in the CloudData Used in This Book
Spark’s Basic ArchitectureSpark ApplicationsSpark’s Language APIsSpark’s APIsStarting SparkThe SparkSessionDataFramesPartitionsTransformationsLazy EvaluationActionsSpark UIAn End-to-End ExampleDataFrames and SQLConclusion
Running Production ApplicationsDatasets: Type-Safe Structured APIsStructured StreamingMachine Learning and Advanced AnalyticsLower-Level APIsSparkRSpark’s Ecosystem and PackagesConclusion
DataFrames and DatasetsSchemasOverview of Structured Spark TypesDataFrames Versus DatasetsColumnsRowsSpark TypesOverview of Structured API ExecutionLogical PlanningPhysical PlanningExecutionConclusion
SchemasColumns and ExpressionsColumnsExpressionsRecords and RowsCreating RowsDataFrame TransformationsCreating DataFramesselect and selectExprConverting to Spark Types (Literals)Adding ColumnsRenaming ColumnsReserved Characters and KeywordsCase SensitivityRemoving ColumnsChanging a Column’s Type (cast)Filtering RowsGetting Unique RowsRandom SamplesRandom SplitsConcatenating and Appending Rows (Union)Sorting RowsLimitRepartition and CoalesceCollecting Rows to the DriverConclusion
Where to Look for APIsConverting to Spark TypesWorking with BooleansWorking with NumbersWorking with StringsRegular ExpressionsWorking with Dates and TimestampsWorking with Nulls in DataCoalesceifnull, nullIf, nvl, and nvl2dropfillreplaceOrderingWorking with Complex TypesStructsArrayssplitArray Lengtharray_containsexplodeMapsWorking with JSONUser-Defined FunctionsConclusion
Aggregation FunctionscountcountDistinctapprox_count_distinctfirst and lastmin and maxsumsumDistinctavgVariance and Standard Deviationskewness and kurtosisCovariance and CorrelationAggregating to Complex TypesGroupingGrouping with ExpressionsGrouping with MapsWindow FunctionsGrouping SetsRollupsCubeGrouping MetadataPivotUser-Defined Aggregation FunctionsConclusion
Join ExpressionsJoin TypesInner JoinsOuter JoinsLeft Outer JoinsRight Outer JoinsLeft Semi JoinsLeft Anti JoinsNatural JoinsCross (Cartesian) JoinsChallenges When Using JoinsJoins on Complex TypesHandling Duplicate Column NamesHow Spark Performs JoinsCommunication StrategiesConclusion
The Structure of the Data Sources APIRead API StructureBasics of Reading DataWrite API StructureBasics of Writing DataCSV FilesCSV OptionsReading CSV FilesWriting CSV FilesJSON FilesJSON OptionsReading JSON FilesWriting JSON FilesParquet FilesReading Parquet FilesWriting Parquet FilesORC FilesReading Orc FilesWriting Orc FilesSQL DatabasesReading from SQL DatabasesQuery PushdownWriting to SQL DatabasesText FilesReading Text FilesWriting Text FilesAdvanced I/O ConceptsSplittable File Types and CompressionReading Data in ParallelWriting Data in ParallelWriting Complex TypesManaging File SizeConclusion
What Is SQL?Big Data and SQL: Apache HiveBig Data and SQL: Spark SQLSpark’s Relationship to HiveHow to Run Spark SQL QueriesSpark SQL CLISpark’s Programmatic SQL InterfaceSparkSQL Thrift JDBC/ODBC ServerCatalogTablesSpark-Managed TablesCreating TablesCreating External TablesInserting into TablesDescribing Table MetadataRefreshing Table MetadataDropping TablesCaching TablesViewsCreating ViewsDropping ViewsDatabasesCreating DatabasesSetting the DatabaseDropping DatabasesSelect Statementscase…when…then StatementsAdvanced TopicsComplex TypesFunctionsSubqueriesMiscellaneous FeaturesConfigurationsSetting Configuration Values in SQLConclusion
When to Use DatasetsCreating DatasetsIn Java: EncodersIn Scala: Case ClassesActionsTransformationsFilteringMappingJoinsGrouping and AggregationsConclusion
What Are the Low-Level APIs?When to Use the Low-Level APIs?How to Use the Low-Level APIs?About RDDsTypes of RDDsWhen to Use RDDs?Datasets and RDDs of Case ClassesCreating RDDsInteroperating Between DataFrames, Datasets, and RDDsFrom a Local CollectionFrom Data SourcesManipulating RDDsTransformationsdistinctfiltermapsortRandom SplitsActionsreducecountfirstmax and mintakeSaving FilessaveAsTextFileSequenceFilesHadoop FilesCachingCheckpointingPipe RDDs to System CommandsmapPartitionsforeachPartitionglomConclusion
Key-Value Basics (Key-Value RDDs)keyByMapping over ValuesExtracting Keys and ValueslookupsampleByKeyAggregationscountByKeyUnderstanding Aggregation ImplementationsOther Aggregation MethodsCoGroupsJoinsInner JoinzipsControlling PartitionscoalescerepartitionrepartitionAndSortWithinPartitionsCustom PartitioningCustom SerializationConclusion
Broadcast VariablesAccumulatorsBasic ExampleCustom AccumulatorsConclusion
The Architecture of a Spark ApplicationExecution ModesThe Life Cycle of a Spark Application (Outside Spark)Client RequestLaunchExecutionCompletionThe Life Cycle of a Spark Application (Inside Spark)The SparkSessionLogical InstructionsA Spark JobStagesTasksExecution DetailsPipeliningShuffle PersistenceConclusion
Writing Spark ApplicationsA Simple Scala-Based AppWriting Python ApplicationsWriting Java ApplicationsTesting Spark ApplicationsStrategic PrinciplesTactical TakeawaysConnecting to Unit Testing FrameworksConnecting to Data SourcesThe Development ProcessLaunching ApplicationsApplication Launch ExamplesConfiguring ApplicationsThe SparkConfApplication PropertiesRuntime PropertiesExecution PropertiesConfiguring Memory ManagementConfiguring Shuffle BehaviorEnvironmental VariablesJob Scheduling Within an ApplicationConclusion
Where to Deploy Your Cluster to Run Spark ApplicationsOn-Premises Cluster DeploymentsSpark in the CloudCluster ManagersStandalone ModeSpark on YARNConfiguring Spark on YARN ApplicationsSpark on MesosSecure Deployment ConfigurationsCluster Networking ConfigurationsApplication SchedulingMiscellaneous ConsiderationsConclusion
The Monitoring LandscapeWhat to MonitorDriver and Executor ProcessesQueries, Jobs, Stages, and TasksSpark LogsThe Spark UISpark REST APISpark UI History ServerDebugging and Spark First AidSpark Jobs Not StartingErrors Before ExecutionErrors During ExecutionSlow Tasks or StragglersSlow AggregationsSlow JoinsSlow Reads and WritesDriver OutOfMemoryError or Driver UnresponsiveExecutor OutOfMemoryError or Executor UnresponsiveUnexpected Nulls in ResultsNo Space Left on Disk ErrorsSerialization ErrorsConclusion
Indirect Performance EnhancementsDesign ChoicesObject Serialization in RDDsCluster ConfigurationsSchedulingData at RestShuffle ConfigurationsMemory Pressure and Garbage CollectionDirect Performance EnhancementsParallelismImproved FilteringRepartitioning and CoalescingUser-Defined Functions (UDFs)Temporary Data Storage (Caching)JoinsAggregationsBroadcast VariablesConclusion
What Is Stream Processing?Stream Processing Use CasesAdvantages of Stream ProcessingChallenges of Stream ProcessingStream Processing Design PointsRecord-at-a-Time Versus Declarative APIsEvent Time Versus Processing TimeContinuous Versus Micro-Batch ExecutionSpark’s Streaming APIsThe DStream APIStructured StreamingConclusion
Structured Streaming BasicsCore ConceptsTransformations and ActionsInput SourcesSinksOutput ModesTriggersEvent-Time ProcessingStructured Streaming in ActionTransformations on StreamsSelections and FilteringAggregationsJoinsInput and OutputWhere Data Is Read and Written (Sources and Sinks)Reading from the Kafka SourceWriting to the Kafka SinkHow Data Is Output (Output Modes)When Data Is Output (Triggers)Streaming Dataset APIConclusion
Event TimeStateful ProcessingArbitrary Stateful ProcessingEvent-Time BasicsWindows on Event TimeTumbling WindowsHandling Late Data with WatermarksDropping Duplicates in a StreamArbitrary Stateful ProcessingTime-OutsOutput ModesmapGroupsWithStateflatMapGroupsWithStateConclusion
Fault Tolerance and CheckpointingUpdating Your ApplicationUpdating Your Streaming Application CodeUpdating Your Spark VersionSizing and Rescaling Your ApplicationMetrics and MonitoringQuery StatusRecent ProgressSpark UIAlertingAdvanced Monitoring with the Streaming ListenerConclusion
A Short Primer on Advanced AnalyticsSupervised LearningRecommendationUnsupervised LearningGraph AnalyticsThe Advanced Analytics ProcessSpark’s Advanced Analytics ToolkitWhat Is MLlib?High-Level MLlib ConceptsMLlib in ActionFeature Engineering with TransformersEstimatorsPipelining Our WorkflowTraining and EvaluationPersisting and Applying ModelsDeployment PatternsConclusion
Formatting Models According to Your Use CaseTransformersEstimators for PreprocessingTransformer PropertiesHigh-Level TransformersRFormulaSQL TransformersVectorAssemblerWorking with Continuous FeaturesBucketingScaling and NormalizationStandardScalerWorking with Categorical FeaturesStringIndexerConverting Indexed Values Back to TextIndexing in VectorsOne-Hot EncodingText Data TransformersTokenizing TextRemoving Common WordsCreating Word CombinationsConverting Words into Numerical RepresentationsWord2VecFeature ManipulationPCAInteractionPolynomial ExpansionFeature SelectionChiSqSelectorAdvanced TopicsPersisting TransformersWriting a Custom TransformerConclusion
Use CasesTypes of ClassificationBinary ClassificationMulticlass ClassificationMultilabel ClassificationClassification Models in MLlibModel ScalabilityLogistic RegressionModel HyperparametersTraining ParametersPrediction ParametersExampleModel SummaryDecision TreesModel HyperparametersTraining ParametersPrediction ParametersRandom Forest and Gradient-Boosted TreesModel HyperparametersTraining ParametersPrediction ParametersNaive BayesModel HyperparametersTraining ParametersPrediction ParametersEvaluators for Classification and Automating Model TuningDetailed Evaluation MetricsOne-vs-Rest ClassifierMultilayer PerceptronConclusion
Use CasesRegression Models in MLlibModel ScalabilityLinear RegressionModel HyperparametersTraining ParametersExampleTraining SummaryGeneralized Linear RegressionModel HyperparametersTraining ParametersPrediction ParametersExampleTraining SummaryDecision TreesModel HyperparametersTraining ParametersExampleRandom Forests and Gradient-Boosted TreesModel HyperparametersTraining ParametersExampleAdvanced MethodsSurvival Regression (Accelerated Failure Time)Isotonic RegressionEvaluators and Automating Model TuningMetricsConclusion
Use CasesCollaborative Filtering with Alternating Least SquaresModel HyperparametersTraining ParametersPrediction ParametersExampleEvaluators for RecommendationMetricsRegression MetricsRanking MetricsFrequent Pattern MiningConclusion
Use CasesModel Scalabilityk-meansModel HyperparametersTraining ParametersExamplek-means Metrics SummaryBisecting k-meansModel HyperparametersTraining ParametersExampleBisecting k-means SummaryGaussian Mixture ModelsModel HyperparametersTraining ParametersExampleGaussian Mixture Model SummaryLatent Dirichlet AllocationModel HyperparametersTraining ParametersPrediction ParametersExampleConclusion
Building a GraphQuerying the GraphSubgraphsMotif FindingGraph AlgorithmsPageRankIn-Degree and Out-Degree MetricsBreadth-First SearchConnected ComponentsStrongly Connected ComponentsAdvanced TasksConclusion
What Is Deep Learning?Ways of Using Deep Learning in SparkDeep Learning LibrariesMLlib Neural Network SupportTensorFramesBigDLTensorFlowOnSparkDeepLearning4JDeep Learning PipelinesA Simple Example with Deep Learning PipelinesSetupImages and DataFramesTransfer LearningApplying Popular ModelsConclusion
PySparkFundamental PySpark DifferencesPandas IntegrationR on SparkSparkRsparklyrConclusion
Spark PackagesAn Abridged List of Popular PackagesUsing Spark PackagesExternal PackagesCommunitySpark SummitLocal MeetupsConclusion

Content preview from Spark: The Definitive Guide

Chapter 24. Advanced Analytics and Machine Learning Overview

Thus far, we have covered fairly general data flow APIs. This part of the book will dive deeper into some of the more specific advanced analytics APIs available in Spark. Beyond large-scale SQL analysis and streaming, Spark also provides support for statistics, machine learning, and graph analytics. These encompass a set of workloads that we will refer to as advanced analytics. This part of the book will cover advanced analytics tools in Spark, including:

Preprocessing your data (cleaning data and feature engineering)
Supervised learning
Recommendation learning
Unsupervised engines
Graph analytics
Deep learning

This chapter offers a basic overview of advanced analytics, some example use cases, and a basic advanced analytics workflow. Then we’ll cover the analytics tools just listed and teach you how to apply them.

Warning

This book is not intended to teach you everything you need to know about machine learning from scratch. We won’t go into strict mathematical definitions and formulations—not for lack of importance but simply because it’s too much information to include. This part of the book is not an algorithm guide that will teach you the mathematical underpinnings of every available algorithm nor the in-depth implementation strategies used. The chapters included here serve as a guide for users, with the purpose of outlining what you need to know to use Spark’s advanced analytics APIs.