Adding a unique constraint to a Django model
Adding a unique constraint to a Django model with lots of data when using Postgres
Optimizing Development vs. Maintenance Costs
Maintenance may take up to 80% of the total cost of applications during its life cycle. In practice, the actual percentage depends heavily on many things, especially on the life cycle of the system. It may be useful to optimize the development costs for some systems, and maintenance for others.
Master Data and Façades
Lots of Master Data is locked in legacy systems. We need a migration path to release the data.
Ordering, Transactions and Exactly-Once Semantics in Data Integrations
There are three important aspects of robust data integrations when using queues: global ordering of messages, multi-message transactionality, and exactly-once semantics.
Communication Patterns In Data Integrations
Data integrity is especially important in data integrations, whether you use batch or stream processing to deliver data. There are multiple communication patterns available, and only some of them are useful for real-time integrations. All of them have some caveats regarding data integrity.
Lost Integrity On Data Integrity
It seems we have lost our integrity on data integrity. We build systems mostly with eventual consistency as the default approach. At best, eventual consistency pushes the data integrity issues to application developers. At worst, data integrity is just ignored and resolved when problems arise. Better approaches have been available at least from the 1970s, but they seem to be ignored in modern software development.
Lightweight Reference Data
Many enterprises have reference data, but it may not be systematically managed. There may be some code sets in data warehouse tables. Some larger business applications may internally some code sets as well. This post describes a lightweight option for storing subset of reference data, called code sets.
Data Integration Architectures
There are several architectural patterns when working with data integrations, with various benefits and drawbacks. Here I discuss the most relevant ones when building real-time data integrations.
Role of Data Modeling in Data Integrations
When working with data integration platforms, it makes sense to think the role that data modeling as well.
Data Integration Methods
Extract-Tranform-Load (ETL) is the traditional method for data integrations. More modern methods are Extract-Load-Transform (ELT). Sometimes APIs may also be used for data integrations. Data streaming has some benefits over all the alternatives.
About Logging
Recording events as they happen, logging, is a very old abstraction. Why is logging so useful abstraction for maintaining data integrity distributed systems?
Is there a better way to certify your organization?
Your customers may demand certifications, such as ISO 9000, ISO 14000, ISO 27000 etc. These projects are expensive and typically require outside consultants. In the end you may have your certificate, but the project probably did not help you to improve your organization. Even the process documents are just lying somewhere, and they are difficult to access. Is there a better way?
Why to Build a Data Integration Platform?
Why to build a data integration platform? You may already have platforms for processing analytical and reporting data, and IPaaS solutions exist as well, This is a series about the leassons learned from building and adopting state-of-the-art streaming data integration platform.
USM - A Simpler Approach to Service Management
Frameworks, such as ITIL, promise you the management of services. This is not sufficient to make your organization customer-driven. New, more simplified ways for approaching service management are emerging.
Some Considerations When Working With Postgres Full-Text Search
There are some things that are useful to take into account when choosing between Postfres Full Text Search (FTS) and external indexing solutions such as ElasticSearch