Elastic Cloud (Elasticsearch Service)

Elastic Search  is being used for two main streams. The first use case is an internal analytics engine for the usage of our services, which is based on logs that are put into Elastic Search  indices to build different dashboards for key executives and developers, providing different levels of information. This is essential to provide statistics as a nonprofit organization funded by the Department of Energy and other infrastructures. The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast  and another on the West Coast , serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

The second use case is Application Performance Monitoring , where Elastic Search APM  stack is used to collect application performance metrics, primarily using Java, with a bit of Python and Node.js. Those three agents are used along with a standard infrastructure with the APM server that injects everything into Elastic Search indices for incident recovery and finding performance bottlenecks. As a nonprofit organization using an open-source license, there have been no problems with Elastic Search trying to change the license. Since no commercialized services are provided, the organization remains out of the scope of those issues and continues using open-source licenses. Recently, integration with an internal Keycloak instance was completed to provide role-based access to the Kibana application, which was a bit non-trivial but was managed successfully.

The experience regarding the relevancy of search results with Elastic Search is positive since it is used for providing search features for end-users of the Protein Data Bank. During ETL processes, information is collected from different data sources regarding proteins, including protein annotations and structures, which are transformed and loaded into the internal database. One part of that database includes Elastic Search indices. For search capabilities, full-text search is performed for end-users while keyword search is used primarily for internal needs, and no complaints have been heard about either of them.

The main focus is on web access to the Protein Data Bank for scientists and bioinformaticians with a publicly facing service supporting roughly 15 million users and an average load of about 700 requests per second. There are two data centers, one on the East Coast and another on the West Coast, serving the same publicly available interface. Logs from these services are monitored and collected, then put into Elastic Search database, from which different perspectives are provided for various stakeholders.

There are a couple of improvements that would definitely save a lot of headache with Elastic Search. One would be if the open-source license had multi-user access to Kibana, which exists in the paid tier license. There were also some difficult times with parallel and point-in-time interfaces, so better documentation could help, particularly more example-driven content. The provided documentation tends to have some common words but lacks real applicable examples. More specific examples, such as step-by-step guides, would be ideal. From a technical point of view, there are no significant issues recalled as Elastic Search has been absolutely awesome for this use case and covers 100% of the needs.

Elastic Search has been used for roughly five years.

Regarding stability, there are no major incidents recalled with Elastic Search. While not part of the DevOps team, nothing significant has ever exploded to affect the whole organization. If there were issues, the DevOps team was able to fix them quickly. Problems have been experienced with other services, but not with Elastic Search.

In terms of scalability, Elastic Search is good for this organization. A standard three-node setup with multiple clusters is being used for internal and public needs, resulting in six nodes per database across the data centers.

There has been no need to contact customer tech support for Elastic Search. It has been sufficient to visit conferences such as SCALE in Southern California Linux Expo, where Elastic Search has a booth to talk to their staff. The organization often relies on publicly available resources such as forums, issue trackers, and an internal knowledge base. Once, a ticket was created on GitHub  concerning a Kibana issue with Application Performance Monitoring, but that was essentially the extent of it. The main sources of support are conferences and documentation.

No alternatives similar to Elastic Search have been tried. When the discussion about the open-source license started, OpenSearch  was briefly looked into but the decision was made not to move forward because the organization felt secure in the current usage without commercialization.

Elastic Search AI, RAG, and semantic search have not been explored yet, as those opportunities for integration are just beginning. Nothing has been moved into production, so further comment cannot be provided. Standard agents from APM are being used to collect telemetry metrics and send them to the Application Performance Monitoring server, which are different from AI agents.

It is difficult to assess the current pricing of Elastic Search because the organization is in a specific niche as a nonprofit organization. On-premises instances are managed internally and a managed option had been considered, but that did not pass the board's approval. Open-source licensing has worked well, and there have been no ceilings where payment options for additional services needed to be considered. Users are quite satisfied with what is provided, and the organization is happy with what is received from Elastic Search.

The learning curve with Elastic Search was very easy. With a strong background in Java and software engineering, and having a great tutor in the organization who showed how to perform ingestion pipelines with Grok  and how to use the development environment within the stack, the process was manageable. While it might be difficult for middle-level and junior developers, having someone experienced in the organization makes it manageable to share knowledge.

Elastic Search mostly requires maintenance during upgrades. While it is running in standard mode, there have been no major incidents from memory, so it has quite low maintenance requirements.

There are no official partnerships with Elastic Search; the organization is just a user utilizing the open-source license. Overall, this review has been given a rating of 9.

Elastic Search  is used as an observability tool and logging analyzer for solutions that already exist in the company, mainly in FinTech products and financial products.

Elastic Search 's main advantages are the visuals that represent and visualize all entities and system components in a simplified diagram, which provides the ability to identify which component in the system has an issue.

The main benefits include having one centralized place that gathers and aggregates all logs related to different or distributed systems.

Elastic Search could be enhanced by incorporating low-code or no-code plugins that permit developers to integrate it with different or distributed systems. This would allow for configurations that already exist but need customization through plugins or simple code that can facilitate user control over parts of the visuals, dashboards, and sensors.

Graphs should be more interactive by importing different graph schemes or visuals from external resources into Elastic Search.

Given that the product has not been used since 2023, the data might be outdated. If Elastic Search is not integrated with any promised LLM, it should have this capability as soon as possible.

Elastic Search has been used since 2018 to the present moment, depending on the different companies that have been worked with.

Elastic Search is a very stable product, especially after obtaining support licenses from Elastic.

The scalability aspect is straightforward. With self-hosting, resources can be expanded vertically, which is managed from the organization's side.

There is no knowledge about general customer service, but there is previous experience in submitting support cases to the Elastic team to get answers and fulfill requirements.

Elastic Search was installed one time but the work was not completed with it.

Experience exists with Dynatrace  observability tool, but Dynatrace  is completely different from Elastic Search. Dynatrace is comparable to other observability tools in this category.

Elastic Search has been installed in multiple organizations, including the current employer and previous ones, and used for different purposes.

The setup is somewhat complicated due to multiple dependencies and relations with different systems. However, any engineer should be able to understand and read the documentation well to implement it properly based on business needs and requirements.

The implementation team was involved in the deployment.

Return on investment was achieved more than a year ago.

DataDog might be an equivalent product to Elastic Search, though this requires verification.

Hybrid observability was not used. Enterprise API, whether referring to ESB, API Gateway, or middleware, was not used. Serverless  interaction with Kibana was not used. The overall rating for this review is 9 out of 10.

Another feature is image vector analysis, which can authenticate images to prevent impersonation frauds in the ecosystem. This is a major use case in personal information and identifiable information portfolio.

I'm using Elastic Search as an observability tool and a SIEM  tool. The indexing, searching, fast indexing, alert mechanisms, and BCDR compatibility are pretty smooth with Elastic Search.

On the resourcing part, I have cut off a good amount. While I don't have a concrete percentage to mention precisely, it has reduced resources to some extent.

I can share feedback from the SIEM perspective about Elastic Search, as I had evaluated Elastic Search, LogRhythm , QRadar, and Microsoft.

It totally depends upon the nature of business you are in. For my organization, it was imperative to have image scanning in place and identifying frauds happening with PII. From that perspective, Elastic Search has played a vital role. It has good inbuilt EDR capabilities as well, making it a good-to-go tool.

I rate Elastic Search eight out of ten.

My usual use cases for Elastic Search  are that we are using APM , Application Performance Monitoring . We are using Real User Monitoring, as a RUM. We mostly are using it for application performance monitoring and troubleshooting in that regard. I think that's the main thing we're using Elastic Search  observability for right now. We are considering expanding it also to have some Metric Beats and some other features. When we have more data, we will probably start to try to activate AI within Elastic Search. That's a possibility. The Elastic Search platform that we are using is an on-prem installation. It's not a cloud solution we have. This is because of the criticality and confidentiality of the data we have in Elastic Search.

I don't think there's a specific feature within Elastic Search that I have found the most valuable so far. We are more or less using all the features in one way or the other. Elastic Search has impacted my organization positively as we use it for logging and APM. It's not all systems which are using it yet, but it's gathering momentum because they have more use cases to present to other parts of the organization. They explain how different departments are using it, and then people see that they could also benefit from using it. More departments and their systems start to use Elastic Search as a result.

The documentation for Elastic Search can be challenging if you're not already familiar with the platform. The approach to Elastic Search can be difficult if you haven't been working with it previously. Within the product itself, some features could be more intuitive, where currently you need to know specifically where to find them and how to use them.

I have been working with Elastic Search for more than four years now.

From my perspective, Elastic Search has been very stable. The only thing I'm probably missing is what we call the session replay, some kind of tool within Elastic Search based on the data collected that can make some kind of session replay.

Elastic Search is very scalable. The only issue is some features use a huge amount of storage. You need to be in the forefront to make sure that you have the necessary storage to obtain all the data that you're collecting. They probably have surveillance indicating when storage is running low. The engineering department ensures we have sufficient storage. So far, we don't have any scalability issues regarding hosts sending data or the amount of data we are collecting. The engineering department might say we are over-consuming data, but we haven't received any message saying we have reached the ceiling yet.

I do not often communicate with the technical support of Elastic Search. That's the engineering department's responsibility. If I have an issue, I go to the engineering department, and they have the responsibility to communicate with the supplier of Elastic Search or the producer.

I work with many technical solutions compared to Elastic Search, specifically on observability. We are also looking into AI, which is in an experimental phase in my area. We haven't chosen any specific technology regarding AI. For Elastic Search as it is now, we are not looking into other technology to replace it. I am a chief consultant in my department, but in this regard, I'm mostly a user. The ones who are responsible for the platform are in another department. My experience with configuring relevant searches within the Elastic Search platform is limited as I don't search much within the platform. If I have specific needs, I reach out to get assistance from specialists because they are more familiarized with the system and know exactly how to search for things. For implementation configuration of the system, they are more capable than I am, as I'm more of a user than an engineer on the platform. I would rate Elastic Search an eight out of ten because there's always room for improvement, though from a functionality and price perspective, it could be considered a ten.