存算分离下的数据湖架构-程力.pdf

?存储可靠性存储可靠性提供多副本或者纠删码冗余存储，最高可达12个9?的数据持久性。数据安全数据安全提供传输、存储、访问全链路安全保障，支持多租户权限隔离，支持加密传输，支持数据加密存储。可可提供高达99.95%的服务可用性，为您的业务保驾护航。性能性能提供高达30,000QPS的请求性能；保障客户业务在高并发情况下稳定运行。开放兼容开放兼容提供全兼容行业标杆 AWS S3 的接口，提供 Verraform 等多种生态工具支持。低成本低成本提供低至￥0.01/GB?的深度归档存储，支持数据生命周期管理，进一步降低云存储成本。COSCOS?腾讯云上的数据湖态COS 对象存储数据接入元数据管理 DLFData CatalogAI MLEMRDLCDW BIData IngestETL权限管理数据分析数据开发平台 Wedata数据治理数据开发数据集成资源调度数据湖存储终端数据湖存储终端COSCOS：云原生：serverless架构，免运维；数据共享：通过统一的COS对象存储作为弹性底座，结合三层加速器接入多种生态。结构化数据管理：感知数据Table格式，支持按照Hive Table预热，支持IcebergTable管理等。高性价比：弹性、按需扩容生态支持：支持Hadoop生态，K8S生态等多种生态的部署、运维、鉴权等。向业务向业务场景场景：数据本地性加强数据湖结构化容器化调度COSN File SystemThe Hadoop File System For COSCOS GuardCOS Clientmetadataservicefilesystem operation?对象存储 COS数据AIHPC基因HadoopCompatible可架构智能缓存集群化部署FUSECompatibleNativeFS InterfaceS3?Compatible渲染鉴权认证监控告警志管理COS 加速器元数据加速Data Lake?AZ 1EMR AI ML K8SCOS 对象存储GooseFS（Cache Accelerator）元数据加速器（Metadata Accelerator）数据加速器（Data Accelerator）AZ 2EMR AI ML K8SGooseFS（Cache Accelerator）数据加速器（Data Accelerator）IcebergIceberg计算端加速计算端加速AZ端加速端加速存储端加速存储端加速?COS 对象存储TablesHive MetaStore/DLF MetaStoresub-prefix(Partitions)FilesprefixMetadata Cache（RAM/SSD/HDD）Hive/Iceberg Table 预热预热：Table相关的数据从COS对象存储中，被加载到计算节点Data Cache中。Data Cache（RAM/SSD/HDD）Prefix预热预热：COS对象存储中对应Prefix的Objects，被加载到计算节点Data Cache中。Metadata Cache：元数据被缓存到计算节点MetadataCache中，并通过RocksDB进行持久化。?GooseFS?COS 对象存储Near-Real Time IngestionDatabase ChangesEventsReadOptimizedRealTimeBatchIncremental ProcessingInteractive Real Time QueryGoose FS（Cache Accelerator）?CHDFSCOS 对象存储cosn:/Bucket_1/BU_A/data/GooseFS（Cache Accelerator）ofs:/BU_E/data/#goosefs ns create ns_BU_A cosn:/Bucket_1/BU_A/#goosefs ns create ns_BU_B cosn:/Bucket_1/BU_B/#goosefs ns create ns_BU_C cosn:/Bucket_2/BU_C/#goosefs ns create ns_BU_E ofs:/BU_E/#goosefs ns create ns_BU_F ofs:/BU_F/#goosefs ns create ns_BU_G ofs:/BU_G/gfs:/BU_A/data/gfs:/BU_B/data/gfs:/BU_C/data/gfs:/BU_E/data/gfs:/BU_F/data/gfs:/BU_G/data/cosn:/Bucket_1/BU_B/data/cosn:/Bucket_2/BU_C/data/cosn:/Bucket_3/BU_D/data/ofs:/BU_F/data/ofs:/BU_G/data/ofs:/BU_H/data/#hadoop fs ls gfs:/BU_A/#hadoop fs ls gfs:/BU_E/#hadoop fs ls gfs:/BU_D/#hadoop fs ls gfs:/BU_H/#hadoop fs ls cosn:/Bucket_1/BU_A/#hadoop fs ls ofs:/BU_E/#hadoop fs ls cosn:/Bucket_3/BU_D/#hadoop fs ls ofs:/BU_H/CacheCacheNo CacheInvalid#goosefs table attachdb hive thrift:/HOSTNAME:9083 hive_db_name#goosefs table ls db_name table_name#goosefs table load db_name table_name#goosefs table free db_name table_name#goosefs table stat db_name table_nameCREATE EXTERNAL TABLE sales(cs_sold_time_sk int,cs_ship_date_sk int,cs_bill_customer_sk int,cs_bill_cdemo_sk int,cs_bill_hdemo_sk int,.PARTITIONED BY(cs_sold_date_sk string)LOCATIONcosn:/bucket_1/data/catalog_salesCreate Namespace:Attach Database:Load Table:Free Table:GooseFS（Cache Accelerator）gfs:/sales/data/cosn:/bucket_1/data/catalog_sales#goosefs ns create sales cosn:/bucket_1/data#goosefs table attachdb hive thrift:/HOSTNAME:9083 db_sales#goosefs table load db_sales sales#goosefs table free db_sales sales GooseFS（Cache Accelerator）GooseFS（Cache Accelerator）salescatalog_salessalescatalog_salessalescatalog_salesGooseFS（Cache Accelerator）Detach Database:#goosefs table detachdb db_salesTablesHive Meta Store?Impala Cluster 1Impala Cluster 2HDFS Cluster 1HDFS Cluster 2Short-circuitlocal readsShort-circuitlocal readsNode group 1Node group 2COS 对象存储Distributed loadSpark/HiveFlink?209572524727267296183682041803050001000015000200002500030000350004000045000SA2_增强型SSD云盘-GooseFSIT5_本地NVMeSSD-GooseFSS5_SSD云盘-GooseFS_STD3_本地SATA-HDFSS5-CHDFSS5-COSNTPC-DS 性能测试（秒）数据量：3TB49.8%43.1%29.2%?EMR CoreNodeExecutorGooseFSWorkerExecutorSpark DriverSpark ExecutorGooseFSClientCOS 对象存储Client NodeSpark ContextEMRMaster nodeGooseFS MasterYarn Resource ManagerYarn Node ManagerEMR CoreNodeExecutorGooseFSWorkerSpark ExecutorGooseFSClientYarn Node Manager?TKEWorkerNodePodGooseFSWorkerPodSpark Executor/Presto WorkerGooseFSClientTKEWorkerNodePodSpark Executor/Presto WorkerGooseFSClientPodSpark DriverPodGooseFSWorkerPodSpark ExecutorGooseFSClientPodPresto WorkerGooseFSClientPodPresto CoordinatorTKEMasterNodeSparkOperatorPrestoOperatorCOS 对象存储TKEWorkerNodePodGooseFS Master?GPUWorkerNodeGooseFSWorkerTF WorkerCOS 对象存储Client NodeTF ClientMasternodeGooseFS MasterYarn Resource ManagerYarn Node ManagerWorker NodeTF PSYarn Node ManagerGooseFSFUSEGooseFSClientPOSIXGPUWorkerNodeGooseFSWorkerTF WorkerYarn Node ManagerGooseFSFUSEGooseFSClientPOSIX?AZ 1COS 对象存储元数据加速器（Metadata Accelerator）数据加速器（Data Accelerator）AZ端加速端加速存储端加速存储端加速TKE/kubernetesPodPodGooseFS（Cache Accelerator）模型服务TFservingCPU/GPU流计算平台Pod数据源计算端加速计算端加速模型训练平台Goose ClientGoose Client推荐广告搜索?COS 对象存储300GB x 14=4.2 TBTKEWorkerNodePodGooseFSWorkerPod业务PodGooseFSFUSEGooseFSClientPOSIX14个分区同时读取数据4200GB/300s=112 Gbps通过GooseFS，数据只需要被加载一次5分钟加载完成，需要112Gbps带宽每个分区300GB数据，平均分布在 20个（300节点/14）节点上，每个节点上15GB的数据。GooseFS用MEM（32GB/节点）作为Cache每个节点需要从其他19个节点读取285GB数据，同时，被其他节点读取285GB数据。5分钟加载完成的话：单节点网络带宽需求：285GB/300s=8Gbps1个分区 交换机 总带宽需求：160Gbps1个分区 20 个节点X15+个32 GBMEM300 GBMEM?专线模型训练平台GooseFS（Cache Accelerator）GooseFS（Cache Accelerator）TStor COS 一体机COS 对象存储数据清洗平台?