Big Data Introduction
Understand what big data is, the challenges it poses, the Hadoop ecosystem, and how MapReduce compares to Apache Spark
Big Data Introduction
Big data refers to datasets so large or complex that traditional data processing tools can't handle them efficiently. Apache Spark emerged as a solution that overcomes the limitations of earlier systems like Hadoop MapReduce.
What is Big Data?
Big data is characterized by the 5 V's:
| V | Meaning | Example |
|---|---|---|
| Volume | Massive quantity of data | Petabytes of sensor logs |
| Velocity | Speed of data generation | Millions of tweets per minute |
| Variety | Different data formats | CSV, JSON, images, video |
| Veracity | Data quality and trustworthiness | Noisy sensor readings |
| Value | Business insights from data | Fraud detection patterns |
The 5 V's framework helps organizations determine whether they truly have a "big data" problem versus a regular data problem that traditional databases can solve.
Volume
Organizations generate terabytes to petabytes of data daily. A single jet engine produces several terabytes of data per flight. Social media platforms generate hundreds of terabytes every day. Traditional RDBMS systems hit scalability limits at much lower thresholds.
Velocity
Data streams in at unprecedented speeds. Stock market tickers update in microseconds, IoT sensors report readings every second, and clickstream logs accumulate millions of events per hour. Processing must keep up with ingestion rates.
Variety
Data arrives in structured (database tables), semi-structured (JSON, XML, CSV), and unstructured (text, images, video) formats. Traditional databases excel at structured data but struggle with unstructured content.
Many projects fail because they treat all data the same way. Each format has unique parsing requirements and storage considerations.
Challenges of Big Data
- Storage: Traditional storage systems cannot scale economically to petabytes
- Processing: Single-machine processing takes too long or fails outright
- Data Transfer: Moving large datasets over networks creates bottlenecks
- Fault Tolerance: Hardware failures are common at scale — systems must recover automatically
- Skill Gap: Big data tools require specialized knowledge
The Hadoop Ecosystem
Hadoop was the first widely adopted big data platform. Its ecosystem includes:
HDFS (Hadoop Distributed File System)
HDFS splits files into blocks (default 128 MB) and distributes them across cluster nodes. Each block is replicated (default 3x) for fault tolerance.
File: 1 GB log
Blocks: 8 x 128 MB blocks
Replication: 3 copies of each block
Total storage used: 3 GB
Hadoop MapReduce
MapReduce processes data in two phases:
- Map: Transform each input record into key-value pairs
- Reduce: Aggregate values by key
MapReduce writes intermediate results to disk between the Map and Reduce phases. This disk I/O makes it slow for iterative and interactive workloads.
YARN (Yet Another Resource Negotiator)
YARN manages cluster resources, scheduling applications across nodes. It decouples resource management from the processing framework.
Other Hadoop Components
- Hive: SQL-like interface on top of MapReduce
- HBase: NoSQL columnar database on HDFS
- Sqoop: Data transfer between Hadoop and relational databases
- Flume: Log data ingestion
- Oozie: Workflow scheduling
MapReduce vs Spark
| Aspect | MapReduce | Apache Spark |
|---|---|---|
| Processing Model | Disk-based, two-stage | In-memory, DAG-based |
| Speed | Slow (disk I/O overhead) | Up to 100x faster in-memory |
| API | Java only | Python, Scala, Java, R, SQL |
| Iterative Processing | Poor (writes to disk each iteration) | Excellent (keeps data in memory) |
| Real-time | Batch only | Batch, streaming, interactive |
| Fault Tolerance | Task re-execution | RDD lineage, task re-execution |
| SQL Support | Hive (separate) | Spark SQL (built-in) |
| ML Support | Mahout (separate) | MLlib (built-in) |
| Graph Processing | Giraph (separate) | GraphX (built-in) |
| Streaming | Not native | Structured Streaming |
Spark's key advantage is keeping data in memory across operations, which is transformative for iterative algorithms (machine learning) and interactive querying.
When to Choose Spark over MapReduce
- Iterative algorithms: ML training needs multiple passes over data
- Interactive analytics: Ad-hoc queries requiring fast response times
- Streaming workloads: Real-time data processing
- Multi-workload pipelines: Combining SQL, ML, and streaming in one application
When MapReduce Still Makes Sense
- Extremely large datasets exceeding cluster memory
- Low-cost clusters with limited RAM
- Legacy systems already invested in the Hadoop ecosystem
The Spark Advantage
Apache Spark provides a unified analytics engine for:
- Batch processing (DataFrames, RDDs)
- Streaming (Structured Streaming)
- Machine learning (MLlib)
- Graph processing (GraphX)
- SQL analytics (Spark SQL)
All from a single API, eliminating the need to stitch together separate tools.
Spark does not include its own storage system. It reads from and writes to HDFS, S3, GCS, Azure Blob Storage, Cassandra, HBase, JDBC sources, and more.
Use Case: Log Analysis Pipeline
A typical big data pipeline using Spark:
Raw Logs (HDFS/S3)
|
v
Spark Batch/Streaming (cleaning, parsing)
|
v
Transformed Data (Parquet on HDFS/S3)
|
v
Spark SQL (ad-hoc analysis)
|
v
BI Dashboard (Tableau, Superset)
Key Takeaways
- Big data is defined by volume, velocity, variety, veracity, and value
- Hadoop provided the foundation but MapReduce is too slow for interactive/iterative workloads
- Spark processes data in-memory using a DAG engine, achieving 10-100x speedups
- Spark unified batch, streaming, SQL, ML, and graph processing in one framework
- Spark reads from external storage — it is a processing engine, not a storage system
Practice Questions
- What are the 5 V's of big data? Explain each one.
- Why is disk I/O a bottleneck in MapReduce but not in Spark?
- How does HDFS achieve fault tolerance for stored data?
- What is the role of YARN in the Hadoop ecosystem?
- List three scenarios where Spark clearly outperforms MapReduce.
- What happens to intermediate data between Map and Reduce in MapReduce?
- Why is Spark particularly well-suited for iterative machine learning algorithms?
- How does Spark's unified approach differ from stitching together separate Hadoop tools?
- What storage systems can Spark read from and write to?
- Give a real-world use case where big data volume, velocity, and variety all apply.