Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
Summary
Sem Sinchenko describes building a graph analytics pipeline on billion-scale graphs using Apache DataFusion, achieving PageRank on graph500-26 with 5 GB and weakly connected components on twitter_mpi with 10 GB. The post emphasizes disk-based processing, bulk scans, and using DataFusion's spillover and join/aggregation capabilities to avoid Spark/GraphFrames for large graphs.