Let’s suppose I have a very simple table, containing one column with data type “string”.
column_name | type
my_column | string
I would like to create an assert statement in an “Assert” processor to check if
my_column is of data type “string”.
How could I do that with Spark SQL? I know about the describe statement, but I can’t seem to get the output of a describe statement to be checked for correctness in an assert statement.
Important: Since I am working with the “Quality App” in ONE DATA Cartography, I cannot use multiple processors. I must put all of the code within one single Assert processor.
EDIT: With the following Syntax, I am able to get the data type for one single column, but I am still not able to verify the data type for this column in the same processor:
describe (SELECT i.my_column FROM inputTable i describe inputTable my_column