Tag: big data

Geospatial datasets are increasingly large, reaching terabyte and even petabyte scale, which can cause many challenges for geospatial analysts and educators–but Amazon AppStream 2.0 can provide some solutions. In this blog post, we walk through how to deploy QGIS, a no cost, open-source geospatial information system (GIS) application used by geospatial analysts, in Amazon AppStream 2.0. We also load an example dataset to demonstrate how desktop GIS application users can access large, cloud-hosted geospatial datasets with high performance by keeping the data and compute components together on the cloud, and streaming the desktop application instead of downloading the data itself.

A research team led by Dr. Beatriz Martinez Lopez at UC Davis supports pig farmers with a data analytics platform that aggregates and analyzes animal health data to diagnose animal viruses and diseases. But this platform was primarily designed for analysts and data scientists. To truly transform animal disease management, Martinez-Lopez wants to put this data analytics tool into the hands of farmers around the world. So the research team is using the scalable, cost-effective tools of the AWS Cloud, along with a research grant letter of support from AWS, to make this optimized platform a reality.

How do we avoid the next global pandemic? For researchers collaborating with the University of British Columbia Cloud Innovation Center (UBC CIC), the answer to that question lies in a massive library of genetic sequencing data. But there is a problem: the data library is so massive that traditional computing can’t comprehensively analyze or process it. So the UBC CIC team collaborated with computational virologists to create Serratus, an open-science viral discovery platform to transform the field of genomics—built on the massive computational power of the Amazon Web Services (AWS) Cloud.

Terabytes of Earth Observation (EO) data are collected each day, quickly leading to petabyte-scale datasets. By bringing these datasets to the cloud, users can use the compute and analytics resources of the cloud to reliably scale with growing needs. In this post, we show you how to set up a Pangeo solution with Kubernetes, Dask, and Jupyter notebooks step-by-step on Amazon Web Services (AWS), to automatically scale cloud compute resources and parallelize workloads across multiple Dask worker nodes.

Maryland Transportation Institute (MTI), an interdisciplinary research and education organization based out of the University of Maryland, focuses on solving complex transportation problems. When COVID-19 hit, MTI was presented with an urgent new problem: the organization was tasked with gathering, processing, and reporting daily transportation data from nearly 65% of the US population. To keep the public safe, they needed more computing power—quickly. They used the AWS Cloud.

In Europe, government agencies and policy makers see the value in using new technology to unlock digital transformation and deliver better, more innovative citizen services. Using data for statistics initiatives, including open data, can help researchers produce innovative products and tools, including visualisation, to inform government officials ahead of making policy decisions that impact their citizens. When it comes to big data, policy makers need to collaborate with researchers to address issues and challenges in using these new data sources. To work toward this goal, Eurostat, the statistical office of the EU, hosted its bi-annual European Big Data Hackathon.

Severe and extreme weather events not only wreak havoc on lives, property and the economy, but the extent of the destruction and devastation left behind can be difficult to map and quantify. Having high resolution imagery of areas devastated by weather events (hurricanes, tornadoes, floods, and etc.) helps to characterize impacts, formulate needed recovery and response activities, support emergency managers in saving lives, and restart the flow of commerce. NOAA’s data plays a critical role in this process. As part of ASDI, we invited Jena Kent from NOAA’s Big Data Program to share how AWS is helping with disaster response by providing access to aerial data and imagery through open data initiatives.

Enlighten IT Consulting, an AWS Managed Service Partner, is the prime integrator for the Department of Defense (DoD) Big Data Platform (BDP). BDP is a fully accredited government-owned platform used for storing, querying, and analyzing cyber data that runs at petabyte scale in AWS GovCloud (US). The BDP has been adopted by multiple defense and federal civilian agencies as the primary security information event management solution (SIEM) due to its ability to support petabyte-scale data ingest, processing, storage, and visualization without the burden of licensing cost or lock-in.

September was monumental for the blog. Thanks to the continuous innovation of our customers, the blog topped 500 posts since launch! Here are the top September stories that helped us get there.

A guest post by John Brady, CISSP, VP Cyber Security/CISO, Financial Industry Regulatory Authority The Financial Industry Regulatory Authority (FINRA) oversees more than 3,900 securities firms with approximately 640,000 brokers. Every day, we watch over nearly 6 billion shares traded in U.S. equities markets—using technology powerful enough to help detect fraud, abuse, and insider trading. In […]