We need to upload a CSV format file as part of project requirement
We were able upload CSV format file under eCommerce in Version 16.11.07.
We are trying to implement the same in version 18.12.05 - there the same upload
module does not work.
jleroux: I missed to check and allow CSV files when all files are allowed.
That now uses:
https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVFormat.html
The default CSV format used to check upload of CSV files is set in the property
csvformat=CSVFormat.DEFAULT in security.properties
Thanks: Sachin for report
OFBIZ-12587 is a definitive solution to prevent any kind of Freemarker exploits.
But it's hard to realise because OFBiz exposes objects, like attributes from the
Servlet scopes. So in the meantime preventing Freemarker interpolation in fields
is a pragmatic solution.
This is an improvement but needs to be backported because it kinda affects
security
For the next demo to work we need to set the host-headers-allowed to
demo-next.ofbiz.apache.org instead of demo-old.ofbiz.apache.org
Thanks to Drew Foulks for handling the Infra part for us
Changes name="upload_file_type_bogus" in EditCategory.ftl and
EditProductContent.ftl to name="up_load_file_type_bogus"
The last line of deniedWebShellTokens in SecuredUpload and similarly in
SecurityUtilTest classes should no have a comma and anti-slash as previous lines.
Also logs when an image contains one of the token in deniedWebShellTokens
Removes non satisfying last line (because images may contain those strings) in
security.properties and SecurityUtilTest.java
In SecuredUpload, renames deniedFileExtensions and deniedWebShellTokens to
respectively getDeniedFileExtensions and getDeniedWebShellTokens
Adds <<",","+",',','+'>> to deniedWebShellTokens as an obviously non satisfying
(because images may contain those strings, I checked) temporary solution before
looking at Freemarker::WhitelistMemberAccessPolicy as suggested by Matei
Thanks to Matei "Mal" Badanoiu for reporting this post-auth vulnerabily
This was already fixed but we can do better by using SecuredUpload::isValidText
Also grabs some tokens to security.properties::deniedWebShellTokens from
DataEvents.java, and adds "alert" even it's harmless as is, as it's often used
by attackers to demonstrate issues, so will earn time.
Though an improvement, I backport as it's related to security
Trivial change in DataServices.java: uses the SecuredUpload import in 2 places
it was missing.
Adds some tokens in security.properties::deniedWebShellTokens
Removes a duplicated htaccess in security.properties::deniedFileExtensions
Last commits put in an issue in DataServices.java and GroovyBaseScript.groovy.
I should have used SecuredUpload::isValidFileName with dataResourceName and
SecuredUpload::isValidFile with objectInfo. This fixes that.
Also fixes formatting checkstyle issues in SecuredUpload class.
Also fixes formatting checkstyle issues in SecuredUpload class and move allow
all and check line length in right places (it's about content)
In security.properties, adds some deniedFileExtensions and improves few comments
(easier to merge later)
In DataServices::createFileNoPerm filters file upload to verify the filename
before the content is checked if necessary
In previous comment, to prevent linked images inside SVG I used xlink:href
That's no good, it must be followed by <<="http">>, this does it. Else a lot
of SVP images would not be loadable. Because xlink:href can be used in other
cases.
I reopened, forgot that you need to set -PjvmArgs, as I commented before:
<<For a 2GB+ file you need to set Xmx to more than the double of the file size,
eg: gradlew ofbiz -PjvmArgs="-Xms1024M -Xmx5048M">>
This also adds information in security.properties:
<<By default (OOTB) OFBiz is protected against Large File Denial of Service>>
After reviewing https://docs.oracle.com/cd/B14099_19/web.1012/b14014/jspxml.htm,
since we use "<jsp:" this removes "scriptlet>", "declaration>" and "expression>".
They are redundant with "<jsp:"; "<jsp:" covers those already.
Also adds eval( for js and shrinks base64_decode to decode, and processbuilder
to process because of OFBIZ 12571
I always test the denied tokens against a lib of images with 33 600 elements (
3.99 GB). If an image contains the token it can't be in the denied list.
The 1st issue was due to use of processbuilder token. It has been added to
deniedWebShellTokens in security.properties by f2cf262 commit for OFBIZ 11948
The tokens function (for js) and class have been added since while browsing
https://github.com/tennc/webshell
As mention the related deniedWebShellTokens TODO comment: "TODO.... to be continued
with known webshell contents... a complete allow list is impossible anyway...
So, later a deeper review of Groovy sandbox possibilities will be done..
Thanks: Y4er for report
This fixes and improves file upload.
Refactor a bit SecuredUpload class and fixes:
Prevent double extensions
Check extensions
Improves getMimeTypeFromFileName() by checking is file exists
In isValidImageFile() finally use isValidText() bypassing encoding
Completely refactor deniedWebShellTokens in security.properties by re-adding
removed tokens in last commit and adding several new ones. I have still to check
encoded, encrypted and obfuscated webshells)
Modifies SecurityUtilTest::webShellTokensTesting accordingly
While at it better format commonsImagingSupportedFormats, deniedFileExtensions
and imagejSupportedFormats properties for legibility.
Check if createAnonFile service is used in GroovyBaseScript.groovy and if a
complete file name is used (file exist) check, using SecuredUpload, extensions
and prevent double extensions, actually check all the file and stop there.
Adds some "Check if a webshell is not uploaded" comments in
ContentManagementServices.java
DataServices.java
ScaleImage.java
FrameImage.java
ImageManagementServices.java
ProductServices.java
HttpRequestFileUpload.java
ProgramExport.groovy
Trivial comments fixes in catalog.properties
Also fixes OFBIZ 12571 (I use here a space because of pending INFRA 22843)
by simply adding processbuilder to deniedWebShellTokens
In SecuredUpload::isValidImageFile I initially used isValidText() and thought
that decoding would be better so finally used isValidTextFile() instead. But
then valid images files are not passing. So this replaces isValidTextFile by
isValidText there.
Also while at it removes few other PHP tokens, now useless (hopefully, I have
still to check encoded and encrypted PHP webshells), from
security::deniedWebShellTokens. The less tokens we have the better the whole is
legible.
Improves related comments.
Modifies SecurityUtilTest::webShellTokensTesting accordingly
I forgot about SecurityUtilTest::webShellTokensTesting. This fixes it.
Note that I expect to simplify and remove more tokens for PHP, but I have 1st
other things to do...
Lion Tree <liontree0110@gmail.com> has reported us that
"CVE-2020-1938 is not fully fixed".
Though it was fixed by OFBIZ-11407, it still possible for an authenticated user
to upload a webshell included in an image using one of the upload possibilities
in OFBiz. That is not new and covered by OFBIZ-12080 "Secure the uploads", but
was still incomplete.
This enforces the secured uploads by
* checking in SecuredUpload::isValidImageFile that a webshell is not embedded in
an image.
* Keeping only "<%" as a denied token for JSP webshells, instead of currently
"<%@ page"
* Adds "application/text/x-ruby" to SecuredUpload::isExecutable
Also
* Adds "<jsp", and "<?" for PHP. Even if OFBiz does not use PHP at all,
it's often installed on servers.
* Removes "import=\"java" and "runtime.getruntime().exec(". They are no
longer useful since "<%" and "<jsp" block them.
* Remove php token since I'll put "<?" in.
* Adds "#!", rather than adding other shebangs like perl,python and ruby
This will make deniedWebShellTokens more understandable.
But I'm conscious that despite SecuredUpload::isExecutableI I still need to
better handle encoded webshells. I'll do that soon in a second approach.
I'll also certainly more prune PHP related tokens.
Thanks: Lion Tree for report
Despite by best efforts in SecuredUpload::isValidFile and
PricatParseExcelHtmlThread::storePricatFile SpotBugs still reports possible null
dereferencements.
The error is NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE. It's false reports to me.
Hence excluding them in these places
Improve readability using stream api, while adding unit test for
refactoring validation.
Fix deniedWebShellTokens property where some blank space where removed
and '// python' comment was inlined.
This much increases the current security by creating SecuredUpload::isValidText
and call it from ProgramExport.groovy
This said I agree with thiscodecc that a better solution would be to use a
Groovy sandbox, and not only in ProgramExport.groovy.
I had a quick glance at a such solution. Unfortunately as Cédric Champeau reports
at https://melix.github.io/blog/2015/03/sandboxing.html and unlike thiscodecc
suggest there are no "mature solutions on the market".
Nevertheless I'll have a deeper look at an OFBiz specific Groovy sandbox
solution.
Somehow related: I'll also soon extract the list of words used in
SecuredUpload::isValidText in a deniedWebShellWords property in security.properties
Thanks: thiscodecc for report
Fixes a number of issues I spotted while working on OFBIZ-12305 in relation with
OFBIZ-12055
The last change I made for OFBIZ-12305 was incomplete, the files could not be
checked by SecuredUpload because they did not exist! This concerns
ImageManagementServices, DataServices, FrameImage and ProductServices classes.
I used Files::createTempFile to fix that.
Also fixes a bug in SecuredUpload where I reversed the check on
fileToCheck.length in Windows case. I also added a comment, Windows 10 now
allows more length (need an OS parameter change though)
Finally, creates public SecuredUpload::isValidText to be used in OFBIZ-12305
Safer completing solution as recommended by weinull orz <weinull@outlook.com>
Previous commit was OK, but better be safe than sorry ;)
Also forgot to commit the DENIEDFILEEXTENSIONS checkstyle change last time
Thanks: weinull orz for the the right suggestion (done 2021-08-13)
The fix I did is two folds:
filters extensions (thanks to Zhujie's suggestion of a list of extensions to ban)
deletes bad files at the right place (thanks to thiscodecc's report)
Thanks: thiscodecc for the security report
This prevented to run the stable demo, hence the next, weirdly the trunk had
not this problem. Anyway better safe than sorry... Maybe related to Gradle
version or rather libs loaded?
The reason why the stable (and old) demos were not running is because the
EntitySaxReaderTests did not pass