[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-aws-logging-opensearch-s3-centralized-platform-en":3,"article-related-aws-logging-opensearch-s3-centralized-platform-en":31,"series-industry-e9f60ab8-463b-47c5-a7b7-df6c8f44ae92":74},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":23,"views":27,"created_at":28,"published_at":29,"topic_cluster_id":30},"e9f60ab8-463b-47c5-a7b7-df6c8f44ae92","aws-logging-opensearch-s3-centralized-platform-en","AWS logging should be split between OpenSearch and S3","\u003Cp data-speakable=\"summary\">Centralized logging on \u003Ca href=\"\u002Ftag\u002Faws\">AWS\u003C\u002Fa> works best when OpenSearch handles live search and S3 handles retention.\u003C\u002Fp>\u003Cp>Centralized logging on AWS should be split across OpenSearch for hot queries and S3 for archive, not forced into one system.\u003C\u002Fp>\u003Cp>That position is not theory. In the Anblicks architecture, Fluent Bit on EKS streams logs in parallel to Amazon OpenSearch for real-time troubleshooting and Amazon S3 for long-term retention, with Athena reserved for historical SQL queries. That design matches how incidents actually unfold: engineers need fast search during an outage, but they also need cheap retention for audits, forensics, and trend analysis after the fact. Trying to make one storage tier do both jobs turns logging into a cost problem before it becomes an observability problem.\u003C\u002Fp>\u003Ch2>The first argument: hot and cold logs have different jobs\u003C\u002Fh2>\u003Cp>OpenSearch is built for speed, not permanence. When a production issue lands, the value is in being able to search recent logs instantly, filter by service, and correlate events while the incident is still active. That is exactly why the article places OpenSearch on the real-time path and OpenSearch Dashboards on top of it. The point is simple: if a system is used for live debugging, it should be optimized for low-latency retrieval, not for storing every log line from the last two years.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782896572426-vzha.png\" alt=\"AWS logging should be split between OpenSearch and S3\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>S3 is the opposite. It is cheap, durable, and designed for retention at scale. The article’s guidance to continuously archive logs to S3 is the \u003Ca href=\"\u002Fnews\u002Fgoogles-gemini-live-camera-editing-right-move-en\">right move\u003C\u002Fa> because log volume grows faster than most teams expect. A logging platform that keeps all data indexed in OpenSearch pays for that growth through shard overhead, storage, and cluster management. A split architecture avoids that trap by keeping the expensive search index small and the historical archive inexpensive.\u003C\u002Fp>\u003Ch2>The second argument: Fluent Bit makes the split practical\u003C\u002Fh2>\u003Cp>The strongest part of this design is that Fluent Bit can fan out to both destinations without turning the collector into a bottleneck. The article describes Fluent Bit on EKS as a DaemonSet, which means every node runs a lightweight collector close to the workloads. That matters because centralized logging fails when the collector becomes heavy enough to compete with application traffic. Fluent Bit is small enough to be invisible and flexible enough to send the same stream to OpenSearch and S3 at once.\u003C\u002Fp>\u003Cp>That dual delivery is what turns a good architecture into an operational one. Teams do not have to choose between immediate visibility and long-term retention, because the pipeline serves both. If a security team needs to investigate a login spike from last quarter, Athena can query the S3 archive. If an SRE needs to trace a 5xx burst right now, OpenSearch is already indexed and ready. The platform is not elegant because it is minimal. It is elegant because each layer does one job well.\u003C\u002Fp>\u003Ch2>The counter-argument\u003C\u002Fh2>\u003Cp>The best objection is that split logging adds moving parts. OpenSearch, S3, Athena, Glue, SNS, and Fluent Bit create a system that is more complex than a single managed observability product. A smaller team may prefer a vendor platform that bundles ingestion, search, retention, dashboards, and alerting into one bill and one control plane. That argument is serious, because operational simplicity has real value when the team is tiny or the incident load is low.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782896570620-odth.png\" alt=\"AWS logging should be split between OpenSearch and S3\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>The objection also has a cost angle. Open-source plus AWS is not free just because the software is open-source. Someone still has to manage retention policies, index lifecycle, query patterns, compression, partitions, and alert rules. If the team lacks the discipline to tune those pieces, the architecture can become noisy and expensive in a different way.\u003C\u002Fp>\u003Cp>But the counter-argument fails on the core question: what is the logging system for? If it is for durable operations at scale, then vendor convenience is not enough. The article’s architecture is better because it separates latency-sensitive search from low-cost retention and gives the team control over both. That is not extra complexity for its own sake. It is the minimum complexity required to keep logs useful after the first month of growth. The limit is real for very small teams, but for any organization running EKS, EC2, Lambda, and load balancers together, the split is the correct design.\u003C\u002Fp>\u003Ch2>What to do with this\u003C\u002Fh2>\u003Cp>If you are an engineer, design your pipeline around log age and query intent: keep recent, high-value logs in OpenSearch, archive everything else to S3, and use Athena for historical analysis. If you are a PM or founder, treat logging as infrastructure with a budget and retention policy, not as a dashboard feature. Build for the incident you need to solve now, and for the audit you will need six months later.\u003C\u002Fp>","Centralized logging on AWS works best when OpenSearch handles live search and S3 handles retention.","www.anblicks.com","https:\u002F\u002Fwww.anblicks.com\u002Fblog\u002Fbuilding-a-centralized-logging-platform-on-aws-using-fluent-bit-opensearch-and-s3\u002F",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782896572426-vzha.png","industry","en","5ff3a2f2-8b5a-469d-94bd-4a03af33e2c6",[17,18,19,20,21,22],"Fluent Bit","Amazon OpenSearch","Amazon S3","Athena","EKS","centralized logging",[24,25,26],"Split logging by workload: OpenSearch for live search, S3 for retention.","Use Fluent Bit to fan out logs without overloading nodes.","Keep historical analysis in Athena instead of indexing everything forever.",0,"2026-07-01T09:02:21.835314+00:00","2026-07-01T09:02:21.828+00:00","ade4c496-a6ad-4107-879f-b4394bf47c0f",{"tags":32,"relatedLang":33,"relatedPosts":37},[],{"id":15,"slug":34,"title":35,"language":36},"aws-logging-opensearch-s3-centralized-platform-zh","AWS 日誌應分流到 OpenSearch 與 S3，而不是硬塞進單一平台","zh",[38,44,50,56,62,68],{"id":39,"slug":40,"title":41,"cover_image":42,"image_url":42,"created_at":43,"category":13},"777fb6b4-cb95-4faf-8ba2-c915ec340a22","bootdev-go-course-turns-syntax-into-services-en","Boot.dev’s Go course turns syntax into services","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782908267986-zkta.png","2026-07-01T12:17:23.153094+00:00",{"id":45,"slug":46,"title":47,"cover_image":48,"image_url":48,"created_at":49,"category":13},"17d21a9f-2d64-49c0-8a04-fa24d2fab8c6","suse-openchip-risc-v-eu-sovereign-stack-en","SUSE and Openchip turn RISC-V into an EU stack","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782907407926-u3lb.png","2026-07-01T12:02:56.604284+00:00",{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"5040a23c-22d0-47ab-94a5-e10ca77708cb","risc-v-hobbyists-open-hardware-obsession-en","RISC-V hobbyists are proving open hardware still rewards obsession","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782906473059-5j1x.png","2026-07-01T11:47:21.943456+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"2a50a3e6-3552-4dc4-9774-a062f0593447","microsoft-build-2026-securing-code-agents-models-en","Microsoft Build 2026: Securing code, agents, and models","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782903775971-4vnt.png","2026-07-01T11:02:29.750881+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"2556ac13-b8df-462c-be84-5329736ef75e","pentagon-agent-network-ai-battle-decisions-en","Pentagon’s Agent Network speeds AI battle decisions","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782902875963-uxp2.png","2026-07-01T10:47:22.497964+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"18bc1f11-955c-4b08-aca6-0b3d19d7a3f0","codex-openai-coding-agent-real-work-en","Codex is OpenAI’s coding agent for real work","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1782900170418-bnnh.png","2026-07-01T10:02:23.007076+00:00",[75,80,85,90,95,100,105,110,115,120],{"id":76,"slug":77,"title":78,"created_at":79},"d35a1bd9-e709-412e-a2df-392df1dc572a","ai-impact-2026-developments-market-en","AI's Impact in 2026: Key Developments and Market Shifts","2026-03-25T16:20:33.205823+00:00",{"id":81,"slug":82,"title":83,"created_at":84},"5ed27921-5fd6-492e-8c59-78393bf37710","trumps-ai-legislative-framework-en","Trump's AI Legislative Framework: What's Inside?","2026-03-25T16:22:20.005325+00:00",{"id":86,"slug":87,"title":88,"created_at":89},"e454a642-f03c-4794-b185-5f651aebbaca","nvidia-gtc-2026-key-highlights-innovations-en","NVIDIA GTC 2026: Key Highlights and Innovations","2026-03-25T16:22:47.882615+00:00",{"id":91,"slug":92,"title":93,"created_at":94},"0ebb5b16-774a-4922-945d-5f2ce1df5a6d","claude-usage-diversifies-learning-curves-en","Claude Usage Diversifies, Learning Curves Emerge","2026-03-25T16:25:50.770376+00:00",{"id":96,"slug":97,"title":98,"created_at":99},"69934e86-2fc5-4280-8223-7b917a48ace8","openclaw-ai-commoditization-concerns-en","OpenClaw's Rise Raises Concerns of AI Model Commoditization","2026-03-25T16:26:30.582047+00:00",{"id":101,"slug":102,"title":103,"created_at":104},"b4b2575b-2ac8-46b2-b90e-ab1d7c060797","google-gemini-ai-rollout-2026-en","Google's Gemini AI Rollout Extended to 2026","2026-03-25T16:28:14.808842+00:00",{"id":106,"slug":107,"title":108,"created_at":109},"6e18bc65-42ae-4ad0-b564-67d7f66b979e","meta-llama4-fabricated-results-scandal-en","Meta's Llama 4 Scandal: Fabricated AI Test Results Unveiled","2026-03-25T16:29:15.482836+00:00",{"id":111,"slug":112,"title":113,"created_at":114},"bf888e9d-08be-4f47-996c-7b24b5ab3500","accenture-mistral-ai-deployment-en","Accenture and Mistral AI Team Up for AI Deployment","2026-03-25T16:31:01.894655+00:00",{"id":116,"slug":117,"title":118,"created_at":119},"5382b536-fad2-49c6-ac85-9eb2bae49f35","mistral-ai-high-stakes-2026-en","Mistral AI: Facing High Stakes in 2026","2026-03-25T16:31:39.941974+00:00",{"id":121,"slug":122,"title":123,"created_at":124},"9da3d2d6-b669-4971-ba1d-17fdb3548ed5","cursors-meteoric-rise-pressures-en","Cursor's Meteoric Rise Faces Industry Pressures","2026-03-25T16:32:21.899217+00:00"]