Maintenance may take up to 80% of the total cost of applications during its life cycle. In practice, the actual percentage depends heavily on many things, especially on the life cycle of the system. It may be useful to optimize the development costs for some systems, and maintenance for others.
Lots of Master Data is locked in legacy systems. We need a migration path to release the data.
There are three important aspects of robust data integrations when using queues: global ordering of messages, multi-message transactionality, and exactly-once semantics.
Data integrity is especially important in data integrations, whether you use batch or stream processing to deliver data. There are multiple communication patterns available, and only some of them are useful for real-time integrations. All of them have some caveats regarding data integrity.
It seems we have lost our integrity on data integrity. We build systems mostly with eventual consistency as the default approach. At best, eventual consistency pushes the data integrity issues to application developers. At worst, data integrity is just ignored and resolved when problems arise. Better approaches have been available at least from the 1970s, but they seem to be ignored in modern software development.
Many enterprises have reference data, but it may not be systematically managed. There may be some code sets in data warehouse tables. Some larger business applications may internally some code sets as well. This post describes a lightweight option for storing subset of reference data, called code sets.
There are several architectural patterns when working with data integrations, with various benefits and drawbacks. Here I discuss the most relevant ones when building real-time data integrations.
When working with data integration platforms, it makes sense to think the role that data modeling as well.
Extract-Tranform-Load (ETL) is the traditional method for data integrations. More modern methods are Extract-Load-Transform (ELT). Sometimes APIs may also be used for data integrations. Data streaming has some benefits over all the alternatives.
Recording events as they happen, logging, is a very old abstraction. Why is logging so useful abstraction for maintaining data integrity distributed systems?
Your customers may demand certifications, such as ISO 9000, ISO 14000, ISO 27000 etc. These projects are expensive and typically require outside consultants. In the end you may have your certificate, but the project probably did not help you to improve your organization. Even the process documents are just lying somewhere, and they are difficult to access. Is there a better way?
Why to build a data integration platform? You may already have platforms for processing analytical and reporting data, and IPaaS solutions exist as well, This is a series about the leassons learned from building and adopting state-of-the-art streaming data integration platform.
Frameworks, such as ITIL, promise you the management of services. This is not sufficient to make your organization customer-driven. New, more simplified ways for approaching service management are emerging.
There are some things that are useful to take into account when choosing between Postfres Full Text Search (FTS) and external indexing solutions such as ElasticSearch
