Recently, I have been working on a Recommender System model, and I frequently use NumPy to preprocess data before feeding them into the PyTorch model. I have discovered some problems in NumPy functions, so I am sharing them as my experience and a part of my coding notes.<\/p>\n\n\n\n
When you concatenate an These are my observations. If you find any error or disagree, it is welcome to discuss it below.<\/p>\n","protected":false},"excerpt":{"rendered":" Recently, I have been working on a Recommender System model, and I frequently use NumPy to preprocess data before feeding them into the PyTorch model. I have discovered some problems in NumPy functions, so I am sharing them as my experience and a part of my coding notes. np.datetime64 This is a data type addressing […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"slim_seo":[],"footnotes":""},"categories":[3],"tags":[],"class_list":["post-18","post","type-post","status-publish","format-standard","hentry","category-notes"],"_links":{"self":[{"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/posts\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shawpence.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=18"}],"version-history":[{"count":4,"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/posts\/18\/revisions"}],"predecessor-version":[{"id":69,"href":"https:\/\/shawpence.com\/index.php?rest_route=\/wp\/v2\/posts\/18\/revisions\/69"}],"wp:attachment":[{"href":"https:\/\/shawpence.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=18"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shawpence.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=18"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shawpence.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}np.datetime64<\/code> This is a data type addressing date and time data in a NumPy matrix. However, it looks like this function is quite costly. When indexing on a matrix with np.datetime64<\/code> column(s), the processing will become very slow. Here’s my advice: avoid using it, and try to replace it with some other measures. For example, if you want to do some simple arithmetic on days, you’d better operate integer plus or minus n*24*60*60<\/code> on timestamp; do not convert it to a np.datetime64<\/code> type.<\/p>\n\n\n\nnp.datetime64<\/code> is stored as int64 in memory. If you try to store it as int32<\/code>, it will overflow.<\/p>\n\n\n\nint32<\/code> matrix with an int64<\/code> matrix, int32<\/code> will convert to float64<\/code> automatically, not int64<\/code><\/em><\/strong>, due to the difference in precision and length.<\/p>\n\n\n\nnp.isin<\/code> This is a function try to detect whether an element in a NumPy container exists in another NumPy container. It is useful when we try to do intersection operations between a matrix and a vector, specifically if we have a matrix and we want to filter the matrix by doing an intersection on a column of this matrix with another specified vector. In this scenario, other intersection functions are not applicable, such as np.in1d<\/code> or np.intersect1d<\/code>. However, this is also a costly function. Therefore, the only way I can do now to accelerate this function is to leverage multiple processes.<\/p>\n\n\n\n