Blog

Blog

Communication Patterns In Data Integrations

Communication Patterns In Data Integrations

Data integrity is especially important in data integrations, whether you use batch or stream processing to deliver data. There are multiple communication patterns available, and only some of them are useful for real-time integrations. All of them have some caveats regarding data integrity.

in
Lost Integrity On Data Integrity

Lost Integrity On Data Integrity

It seems we have lost our integrity on data integrity. We build systems mostly with eventual consistency as the default approach. At best, eventual consistency pushes the data integrity issues to application developers. At worst, data integrity is just ignored and resolved when problems arise. Better approaches have been available at least from the 1970s, but they seem to be ignored in modern software development.

in
Lightweight Reference Data

Lightweight Reference Data

Many enterprises have reference data, but it may not be systematically managed. There may be some code sets in data warehouse tables. Some larger business applications may internally some code sets as well. This post describes a lightweight option for storing subset of reference data, called code sets.

in
Data Integration Architectures

Data Integration Architectures

There are several architectural patterns when working with data integrations, with various benefits and drawbacks. Here I discuss the most relevant ones when building real-time data integrations.

in
Role of Data Modeling in Data Integrations

Role of Data Modeling in Data Integrations

When working with data integration platforms, it makes sense to think the role that data modeling as well.

in
Data Integration Methods

Data Integration Methods

Extract-Tranform-Load (ETL) is the traditional method for data integrations. More modern methods are Extract-Load-Transform (ELT). Sometimes APIs may also be used for data integrations. Data streaming has some benefits over all the alternatives.

in
About Logging

About Logging

Recording events as they happen, logging, is a very old abstraction. Why is logging so useful abstraction for maintaining data integrity distributed systems?

in
Is there a better way to certify your organization?

Is there a better way to certify your organization?

Your customers may demand certifications, such as ISO 9000, ISO 14000, ISO 27000 etc. These projects are expensive and typically require outside consultants. In the end you may have your certificate, but the project probably did not help you to improve your organization. Even the process documents are just lying somewhere, and they are difficult to access. Is there a better way?

in
Why to Build a Data Integration Platform?

Why to Build a Data Integration Platform?

Why to build a data integration platform? You may already have platforms for processing analytical and reporting data, and IPaaS solutions exist as well, This is a series about the leassons learned from building and adopting state-of-the-art streaming data integration platform.

in
USM - A Simpler Approach to Service Management

USM - A Simpler Approach to Service Management

Frameworks, such as ITIL, promise you the management of services. This is not sufficient to make your organization customer-driven. New, more simplified ways for approaching service management are emerging.

in
Some Considerations When Working With Postgres Full-Text Search

Some Considerations When Working With Postgres Full-Text Search

There are some things that are useful to take into account when choosing between Postfres Full Text Search (FTS) and external indexing solutions such as ElasticSearch