SCD (𝐒𝐥𝐨𝐰𝐥𝐲 𝐂𝐡𝐚𝐧𝐠𝐢𝐧𝐠 𝐃𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧) in hive

Let's #hive

📌 What is SCD and w𝐡𝐚𝐭 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐭𝐲𝐩𝐞𝐬 𝐨𝐟 #𝐒𝐂𝐃 𝐢𝐧 𝐇𝐢𝐯𝐞?

✔ In the context of Hive (a data warehousing and SQL-like query language built on top of Hadoop), SCD stands for 𝐒𝐥𝐨𝐰𝐥𝐲 𝐂𝐡𝐚𝐧𝐠𝐢𝐧𝐠 𝐃𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧.

✔ Slowly Changing Dimensions are used to track changes to dimensional data over time, allowing you to maintain historical information and analyze data at different points in time.

There are mainly three types of Slowly Changing Dimensions (SCD) in Hive:

✅ 𝐓𝐲𝐩𝐞 1 𝐒𝐂𝐃 (𝐇𝐢𝐬𝐭𝐨𝐫𝐢𝐜𝐚𝐥 𝐎𝐯𝐞𝐫𝐰𝐫𝐢𝐭𝐞):

▪ In Type 1 SCD, when a change occurs in a dimension record, the old record is simply updated with the new data, overwriting the existing values.

▪ This approach does not preserve historical data, and any previous versions of the record are lost.

▪ Type 1 SCD is suitable when historical data is not important, and you only need the latest version of the data.

✅ 𝐓𝐲𝐩𝐞 2 𝐒𝐂𝐃 (𝐇𝐢𝐬𝐭𝐨𝐫𝐢𝐜𝐚𝐥 𝐓𝐫𝐚𝐜𝐤𝐢𝐧𝐠 - 𝐀𝐝𝐝 𝐍𝐞𝐰 𝐑𝐞𝐜𝐨𝐫𝐝):

▪ Type 2 SCD is designed to preserve the history of dimension data by adding a new record whenever a change occurs.

▪ This new record contains the updated data along with an additional attribute, such as a timestamp or version number, to indicate when the change occurred.

▪ This approach maintains historical data but can lead to a larger dimension table over time.

✅ 𝐓𝐲𝐩𝐞 3 𝐒𝐂𝐃 (𝐇𝐢𝐬𝐭𝐨𝐫𝐢𝐜𝐚𝐥 𝐓𝐫𝐚𝐜𝐤𝐢𝐧𝐠 - 𝐀𝐝𝐝 𝐂𝐨𝐥𝐮𝐦𝐧𝐬):

▪ Type 3 SCD involves adding new columns to the existing dimension table to track changes. For example, you might have attributes like "current_value" and "previous_value" to store the current and previous versions of the attribute.

▪ This approach keeps the dimension table relatively compact while still allowing some historical tracking.

▪ However, it is limited in the number of changes it can track compared to Type 2 SCD.

☑ Each SCD type has its advantages and use cases, and the choice of SCD type depends on the specific requirements of your data analysis and reporting needs.